Contradiction-focused qualitative evaluation of textual entailment
نویسندگان
چکیده
In this paper we investigate the relation between positive and negative pairs in Textual Entailment (TE), in order to highlight the role of contradiction in TE datasets. We base our analysis on the decomposition of Text-Hypothesis pairs into monothematic pairs, i.e. pairs where only one linguistic phenomenon at a time is responsible for entailment judgment and we argue that such a deeper inspection of the linguistic phenomena behind textual entailment is necessary in order to highlight the role of contradiction. We support our analysis with a number of empirical experiments, which use current available TE systems.
منابع مشابه
Toward Qualitative Evaluation of Textual Entailment Systems
This paper presents a methodology for a quantitative and qualitative evaluation of Textual Entailment systems. We take advantage of the decomposition of Text Hypothesis pairs into monothematic pairs, i.e. pairs where only one linguistic phenomenon at a time is responsible for entailment judgment, and propose to run TE systems over such datasets. We show that several behaviours of a system can b...
متن کاملA Joint Syntactic-Semantic Representation for Recognizing Textual Relatedness
This paper describes our participation in the Recognizing Textual Entailment challenge (RTE-5) in the Text Analysis Conference (TAC 2009). Following the two-stage binary classification strategy, our focus this year is to recognize related Text-Hypothesis pairs instead of entailment pairs. In particular, we propose a joint syntactic-semantic representation to better capture the key information s...
متن کاملTowards Effective Tutorial Feedback for Explanation Questions: A Dataset and Baselines
We propose a new shared task on grading student answers with the goal of enabling welltargeted and flexible feedback in a tutorial dialogue setting. We provide an annotated corpus designed for the purpose, a precise specification for a prediction task and an associated evaluation methodology. The task is feasible but non-trivial, which is demonstrated by creating and comparing three alternative...
متن کاملUnsupervised Resource Creation for Textual Inference Applications
This paper explores how a battery of unsupervised techniques can be used in order to create large, high-quality corpora for textual inference applications, such as systems for recognizing textual entailment (TE) and textual contradiction (TC). We show that it is possible to automatically generate sets of positive and negative instances of textual entailment and contradiction from textual corpor...
متن کاملWUST at NTCIR-10 RITE-2 Task: Multiple Feature Approach to Chinese Textual Entailment
ABSTRACT This paper describes our work in NTCIR-10 on RITE-2 Binary-class (BC) subtask and Multi-class (MC) subtask in Simplified Chinese. We construct the classification model based on support vector machine to recognize semantic inference in Chinese text pair, including entailment and non-entailment for BC subtask and forward entailment, bidirectional entailment, contradiction and independenc...
متن کامل